Select Elements by Proximity
A common problem while writing instructions that one encounters is interacting with elements that share the same text.
An example can be seen when you interactively annotate your Visual Studio Code Editor. Notice that the magnifying glass icon and many other elements throughout the UI share the same name:
To ensure you’re able to instruct AskUI to select the correct element, the use of relational element-descriptions can be employed. Relational element-descriptions describe the element relative to other elements in the UI.
Selecting an Element by Visual Relation
After reading the next section you will know how to use the full power of all the relational element-descriptions. Additionally, you learn what pitfalls you can fall into and how to avoid them in the future.
We will use the Selectorshub practice page for the demonstration.
We’ll discuss the following relational element-descriptions:
above()
When you want to click on a textfield and it is above an element, like for example a button with the text Submit. You can do it with above(). The following code snippet moves the mouse to the textfield above the Submit-button:
await aui
.click()
.textfield()
.above()
.button()
.withText('Submit')
.exec();
below()
When you want to select a textfield you can do so by finding the correct label, which is often above the textfield. The following code snippet moves the mouse to the textfield below the text Mobile Number:
await aui
.moveMouseTo()
.textfield()
.below()
.text()
.withText('Mobile Number')
.exec();
contains()
For selecting an element, that contains another element, contains() is the right candidate. It is especially useful if you want to select a textfield with a placeholder text inside it. The text inside the textfield is annotated as an element itself.
If you have problems with selecting a specific element, always run annotate() to create a screenshot of all the annotations or use annotateInteractively() to see if you need to use contains().
The following snippet moves the mouse to a textfield based on its placeholder text First Crush which is contained in the textfield:
await aui
.moveMouseTo()
.textfield()
.contains()
.text()
.withText('First Crush')
.exec();
in()
When you want to target an element that is inside another element you can use in().
--------------------
| outerEl |
| -------------- |
| | innerEl | |
| -------------- |
| |
--------------------
The following code snippet moves the mouse pointer to the text of the first textfield AskUI found:
await aui
.moveMouseTo()
.text()
.in()
.textfield()
.exec();
leftOf() and rightOf()
If you want to select an element based on its location left or right of another element you have to use leftOf() or rightOf() respectively.
💡 If you do not specify another element-description like withText() then you will get the nearest element. Otherwise, AskUI retrieves the nearest element that matches the element-description!
await aui
.moveMouseTo()
.text()
.leftOf()
.text()
.withText('Denmark')
.exec();
await aui
.moveMouseTo()
.text()
.rightOf()
.text()
.withExactText('Joe Root')
.exec();
nearestTo()
Filtering with the nearestTo() will return the element nearest to another element. This is useful when the direction is not clear on where to search. Especially responsive designs are prone to wrap elements into a new line where leftOf() and rightOf() would fail.
await aui
.moveMouseTo()
.textfield()
.nearestTo()
.text()
.withTextRegex('User Em*')
.exec();
Additional Considerations About the Distance Metric
The distance is based entirely on physical distance to the bounding box of a specified element for above(), below(), leftOf() and rightOf().
For above() it takes the upper-left and upper right-corner coordinates of the bounding box. The area that is searched is bordered by orthogonal lines upwards from these coordinates. Every element that is touched by that area can be found.
| ------------ |
| | Returned | |
| ------------ |
| |
| | ----------------
| SEARCHED | | Not returned |
| AREA | ----------------
| |
------------------
| button |
------------------
For nearestTo() it also considers special cases, for example, modal dialogues. Therefore the element selected by AskUI might sometimes be wrong from a user's point of view.
You can use moveMouseTo() like in the following example to see what element an instruction targets.
await aui
.moveMouseTo()
.textfield()
.above()
.button()
.withText('Submit')
.exec()